From a viewpoint of performance, it is desirable to reduce as much as possible an overlap between inter-node communication performed by one parallel job and an inter-node communication performed by another parallel job.
Examples of network topologies of a large scale parallel calculation system include a network topology called a mesh or torus. In this system, each communication route of the inter-node communication assigned to a corresponding job is closed to a single sub-mesh or sub-torus so as to avoid interference of communication between the jobs in some cases. However, limiting the node assigned to one job to within the single sub-mesh or the sub-torus is disadvantageous with respect to node assignment efficiency.
In order to increase availability of the node, a plurality of sub-meshes or sub-torus may be used by one job. In this case, interference of communication between jobs occurs.
In the network in which the network topology is the mesh or the torus, a communication route in transfer in which hardware itself designates a start point and an end point of the communication is often determined by a method called dimension-order routing. The dimension-order routing is a kind of static routing, and the route to be used is fixed at a time when the start point and the end point of the communication are specified.
FIG. 23 is a diagram illustrating an example of the dimension-order routing in two dimensions. In FIG. 23, a horizontal direction represents an X axis direction and a vertical direction represents a Y axis direction. A communication packet from a node (A) to a node (B) first proceeds in the X axis direction from coordinates of the node (A), and proceeds in the Y axis direction when an X coordinate becomes equal to a coordinate of the node (B).
In an automatic route selection by hardware, routes other than the route illustrated in FIG. 23 are not used. Therefore, in a system using the dimension-order routing, in a case of reducing interference of communication by using a route different from the route automatically selected by the hardware, software performs the following processes. That is, the software sets a node to be a relay point to nodes other than the start point and the end point of the communication, and explicitly communicates via the node of the relay point. In the following description, the node at the relay point is referred to as a relay node.
FIG. 24 is a diagram illustrating an example of a detour route by relay node setting. The software that performs route determination sets a relay node (C). In addition, the software activates a relay processing function on the relay node (C) or sets up a command sequence for automatic transfer function by hardware on the relay node (C). For example, the automatic transfer function corresponds to InfiniBand CORE-Direct and Tofu session-mode functions.
When communication from the node (A) to the node (B) is performed, the node (A) adds “transfer instruction information to the node (B)” to a header of a communication packet and transmits the communication packet toward the node (C). Accordingly, the communication packet reaches the node (B) by the detour routes (A), (C) and (B).
Generally, in an m-dimensional mesh or torus network, an network address of each node is expressed as a set of m integers such as (x1, . . . , xm), and an upper limit and a lower limit of the integers of each dimension are defined as a1≤x1≤b1, a2≤x2≤b1, . . . , am≤xm≤bm.
Here, a size (hop number) of each dimension of the network is (bi−ai+1)=Li (i=1, 2, . . . , m). A physical network link (hereinafter, simply referred to as “link”) exists between two nodes where coordinates of each dimension differ by one. However, in the case of the torus, there is a link between the nodes of coordinates a1 and b1 in the i-th dimension. In the torus, when a difference between the coordinates of the i-th dimension is calculated by mod(bi×ai+1), handling of the mesh and torus coordinates may be substantially the same. Japanese National Publication of International Patent Application No. 2011-516945 is an example of the related art.